Allow opt-out of just-bash dependencies - #33
Open
aron-cf wants to merge 1 commit into
Open
Conversation
The worker backend embeds the whole just-bash shell in SHELL_MODULES, and a handful of optional commands dominate the upload: curl carries undici (~620 KB) and html-to-markdown carries domino (~555 KB), with python, sqlite, and js-exec behind them. Every consumer paid for all of them because the modules shipped as one frozen record with no seam to drop a command. build-bundle.mjs now partitions the emitted modules into a core group plus one group per optional command and writes them under generated/ as separate modules. A chunk lands in a feature group only when that feature is its sole reacher and core cannot reach it, so shared chunks and anything a kept command needs stay in core and dropping one feature never breaks another. undici is attributed to curl by hand because just-bash loads it from the shell body rather than through the curl command chunk. shell-modules.ts assembles SHELL_MODULES from the groups, importing each through its @cloudflare/computer/shell/<feature> subpath so a consumer can swap one for @cloudflare/computer/empty with a wrangler alias. That drops the feature's exclusive chunks, heavy dependency included, from the uploaded Worker. Leaving the alias out ships the whole shell, byte-for-byte the single-file bundle. The command still parses; invoking it after its chunk is gone fails at runtime with a module-not-found. The worker example drops curl and html-to-markdown, taking its upload from ~5,956 KiB to ~4,774 KiB raw. The package and worker docs describe the mechanism and the available features.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The worker backend embeds the entire just-bash shell in
SHELL_MODULES, a single frozen record ofmoduleName → { js }thatWorkerBackendspreads into the Worker Loader callback. A handful of optional commands dominate that upload —curlpulls in undici (~620 KB) andhtml-to-markdownpulls in domino (~555 KB), withpython,sqlite, andjs-execbehind them — and every consumer paid for all of them because there was no seam to drop a command.This splits the bundle into independently-droppable feature groups so a consumer can remove commands they don't need at build time, without any change to the default output.
What changed
build-bundle.mjsnow partitions esbuild's emitted modules into acoregroup plus one group per optional command, and writes each togenerated/<group>.ts(gitignored, regenerated onprepare/pretest/pretypecheck). Assignment is by reachability, not by name:so dropping one feature can never break another.
undiciis attributed tocurlexplicitly, because just-bash loads it from the shell body rather than through thecurlcommand chunk.A new checked-in
shell-modules.tsreassemblesSHELL_MODULESfrom the groups, importing each through its own published subpath:Because the groups are imported by subpath, a consumer can swap any one for an empty module table with a wrangler
alias, and esbuild then never pulls that group — or its heavy dependency — into the uploaded Worker.Consumer config
Drop a feature in
wrangler.jsonc:Optional features and the rough weight each one removes from the upload when aliased out (generated group source):
curl@cloudflare/computer/shell/curlhtml-to-markdown@cloudflare/computer/shell/html-to-markdownsqlite@cloudflare/computer/shell/sqlitejs-exec@cloudflare/computer/shell/js-execpython@cloudflare/computer/shell/pythonFor reference,
core(never removable) is ~2,346 KiB raw / ~538 KiB gzip. Leaving thealiasout ships the whole shell, byte-for-byte the previous single-file bundle. A dropped command still parses; invoking it after its chunk is gone fails at runtime with a module-not-found.The example Worker drops
curlandhtml-to-markdown, which takes its upload from ~5,956 KiB → ~4,774 KiB raw (~1,344 → ~1,071 KiB gzip). Default build (nothing aliased) is unchanged at ~5,956 KiB (538 KiB gzip).Alternative considered:
defineconstantsA per-feature
defineconstant was considered — one boolean per feature that gates itsimport:This does drop the bytes (verified against esbuild), since esbuild constant-folds the dead branch and tree-shakes the group away — but only when the generated groups are marked
sideEffects: false, and it needs atypeofguard on each constant so features still default on when a consumer sets nothing. It's also still onedefineentry per feature.aliasachieves the same removal with nosideEffectshint and no magic globals, so it's what this PR ships.Follow-ups
This PR only removes optional command chunks;
core(~2,346 KiB raw / ~538 KiB gzip) is unchanged. Dead-code elimination has already run, so what's left is reachable, but a few levers could shrink it further:build-bundle.mjscurrently emits the embedded module sources unminified. Addingminifywould take ~617 KiB raw / ~66 KiB gzip offcore(and ~296/30 offhtml-to-markdown, ~121/11 offcurl) with no functionality lost — the largest single win.awk(a full language interpreter),sed,tar, andgitlive in core today but are the kind of thing many consumers don't need; they could become optional groups on the same alias mechanism.corecarries re2js (the RE2 regex engine, ~225 KiB) for RE2-compatible semantics and pako (~128 KiB, via isomorphic-git). Falling back to nativeRegExpwhere semantics allow, and to Workers'node:zlibfor pako, would remove both.Testing
npm run typecheck— passes.npx biome check .— passes.npx vitest run(package unit tests) — 799 passed;npm run test:worker-backend(real workerd Loader path) — 4 passed.wrangler deploy --dry-runon the example: 5,956 KiB with no alias, 4,774 KiB withcurl+html-to-markdowndropped.Notes
The generated
src/backends/worker/generated/files are gitignored; runnpm run build:shell-bundle --workspace @cloudflare/computerafter applying to regenerate them before building.